Device embedded in, or attached to, a pillow configured for in-bed monitoring of respiration

ABSTRACT

A monitoring device has microphones, an ADC; a digital radio; and a processor with firmware. The firmware includes code for digitizing audio from the microphones into time-domain audio, performing FFT to provide frequency-domain audio, running a first neural network on time domain and frequency-domain audio to extract features, executing a classifier on the features to identify candidate events, and using the digital radio to upload candidate events and features. A pressure sensor awakens the processor from a low-power state. In particular embodiments, the first neural network is an embedded Gated Recurrent Unit having weights trained to extract features of use in the classifier; and candidate events include normal inhalation and exhalation breathing sounds, crackles, wheezes, coughs, snoring, gasping, choking, and speech sounds and in some embodiments heart sounds. A method of monitoring breathing during sleep includes attaching the device to, or embedding the device within, a pillow.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a 35 U.S.C. § 371 filing of InternationalApplication No. PCT/US2019/048826 filed Aug. 29, 2019, which claimspriority to U.S. Provisional Patent Application No. 62/726,146 filedAug. 31, 2018, the entire contents of which are incorporated herein byreference.

BACKGROUND

In the US alone, an estimated 150 million of people suffer from chronicdiseases, including 42 million people suffering from chronic respiratorydiseases (including asthma, chronic bronchitis, chronic obstructivepulmonary disease (COPD), and sleep apnea) and 5.8 million peoplesuffering from heart failure.

The severity of these chronic disease can fluctuate and progress overtime. Between regularly scheduled doctor's visits and tests, patientsoften experience exacerbations and acute decompensations. Theseexacerbations and acute decompensations are common, are traumatic forpatients, can result in hospitalization and even death, and are costlyfor the healthcare system. Monitoring a patient's symptoms and signs canlead to a better understanding of a patient's disease, which can enablebetter disease management, improved healthcare outcomes, and reducedhealthcare costs.

The patient burden of chronic diseases is high, and patients oftenstruggle to adhere to their treatment and monitoring regimen. Monitoringdevices, such as blood-pressure cuffs, peak-flow meters, andspirometers, all require that a patient not only remember to performtheir monitoring but also put in effort to perform their monitoringcorrectly.

Wrist-worn accelerometers, such as provided in “Fitbit”-style devices,and optical plethysmographic devices, offer more passive monitoring andmay be used to provide some monitoring functions during sleep. Thesedevices, however, do not directly monitor breathing.

It is known that lung and/or heart sounds produced by individuals differbetween those produced during normal health and during episodessymptomatic of respiratory and cardiovascular diseases such as asthma,chronic obstructive pulmonary disease (COPD), pneumonia, cystic fibrosisand congestive heart failure; physicians doing physical examinationstypically listen to these lung and heart sounds though a stethoscope.

SUMMARY

In an embodiment, a device configured for monitoring physiologicalsounds includes at least one microphone coupled to an analog-to-digitalconverter (ADC); a digital radio; and a processor configured withfirmware in a memory. The firmware includes machine readable code forusing the ADC to digitize audio from the at least one microphone intodigitized time-domain audio, performing a fast Fourier transform on thedigitized time-domain audio to provide frequency-domain audio, executinga first neural network on the digitized time-domain audio and thefrequency-domain audio to extract features from the audio and at leastone pressure sensor, executing a classifier on the features to identifycandidate events, and using the digital radio to upload the candidateevents and features. The at least one pressure sensor is coupled toawaken the processor from a low-power state. In particular embodiments,the first neural network is an embedded Gated Recurrent Unit (e-GRU)having weights trained to extract features of use in the classifier. Inparticular embodiments, the candidate events include one or more ofnormal inhalation and exhalation breathing sounds, crackles, wheezes,coughs, snoring, gasping, choking, and speech sounds. In particularembodiments, the candidate events include heart sounds.

In an embodiment, a method of monitoring breathing during sleep includesattaching to a pillow, or embedding within a pillow, a breathing monitordevice; extracting features from sound recorded with the breathingmonitor device; classifying the extracted features to detect candidateevents; and uploading the candidate events with the extracted featuresand a timestamp.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of a device for monitoring breathing sleep andadapted to be installed within a pillow.

FIG. 2 is a block diagram of a system incorporating the device of FIG.1.

FIG. 3 is an illustration of the device of FIG. 1 positioned in apillow.

FIG. 4 is a flowchart illustrating a method for monitoring breathingduring sleep.

FIG. 5 is a block diagram illustrating firmware on the device formonitoring breathing during sleep.

DETAILED DESCRIPTION OF THE EMBODIMENTS

A sleep and breathing monitor device 100 (FIG. 1) has a low powerprocessor 102 with a memory 104 containing machine readable instructionsof firmware 106. The device also has at least one, and, in particularembodiments, an array of, microphones 108, 110, 112, 114 orpiezoelectric transducers adapted to monitor breath sounds. Inembodiments, the device also has one or more pressure sensors 116 andaccelerometers 118. The device is powered by a battery 120; battery 120is in a particular embodiment coupled to be recharged by a charger 122and has a battery voltage sensor 123 or charge monitor. The sensors,including battery voltage sensor 123, microphones 108, 110, 112, 114,pressure sensor 116, and accelerometers 118 are coupled through analogfront-end circuitry (not shown) to the processor 102 and are configuredto be read through an analog-to-digital converter 130. Some, but inembodiments not all, of the sensors, including at least one microphone108 and pressure sensor 116, also couple to provide signals to a wakeupcircuit 124 adapted to arouse the processor 102 from a standby state toan operating state when the sensors coupled to it detect conditionsindicating that the sleep and breathing monitor is in use for monitoringsleep. Memory 104 also contains a data store 132 configured to holdsignatures of interesting sleep events, and processor 102 is coupled toa low-powered digital radio transceiver 134 through which the signaturesof interesting sleep events may be transmitted to a smartwireless-equipped device such as a cell phone, tablet computer, or bodyarea network server, for transmission to a server; in a particularembodiment digital radio transceiver 134 is a Bluetooth compatibledigital radio. Processor 102 is also coupled to a clock/timer circuit136.

The microphones 108, 110, 112, 114 are adapted to record breath soundsas well as other physiological sounds such as heart sounds, beats andmurmurs. Firmware 106 is configured to extract features from recordedsounds and to classify those features to determine events potentially ofinterest, these features are stored in data store 132 until they areuploaded.

In an embodiment of a system 200 (FIG. 2) using the sleep and breathingmonitor device 100, 202 herein described, the device 202 periodicallyuses digital radio transceiver 134 to upload the extracted features ofbreathing and other events classified as potentially of interest frommemory 132, in some embodiments with timestamps from clock/timer 136,via a compatible digital radio, such as a Bluetooth radio, into awireless-equipped smart device 204 such as a smartphone, tabletcomputer, cell-capable smart watch, or body area network (BAN) hub. Thesmart device 204 then relays these extracted features, classifications,and timestamps through a cell phone tower 206 or Wi-Fi hotspot andinternet 208 to a server 210 where event classification and statisticalanalysis code 212 in server memory processes the extracted features andpreliminary classifications to identify events of interest and compilesrelevant statistics. The events of interest and relevant statistics mayin embodiments include length and duration of sleep, frequency andlength of cessation of breathing during sleep, snoring, gasping snortingand similar episodes during snoring as may occur during obstructivesleep apnea, wheezing as may occur in asthma or chronic obstructivepulmonary disease (COPD), crackles as may occur in pneumonia orcongestive heart failure, surges in heart rate indicative of stress, andwakeful spells including time and duration of subject arousal such aswhen subjects rise to urinate; these events of interest and statisticsare recorded in database 214. A user or physician may use a smartphone,tablet computer, or workstation 216 through internet 208 and server 210to access database 214 to view logs of these events of interest andstatistics to better diagnose breathing and sleep-related issues with auser or patient. In some embodiments, an optional second sleep andbreathing monitor device 218 may be provided to provide bettermonitoring of one user, or to monitor two users sharing a bed.

The device is either embedded within foam or another pillow material ofa pillow, or designed as an accessory to be attached to a pillow. Forinstance, when configured as an accessory, the device can take the formof an electronic/smart pillow case for use with standard pillows, or asan electronic pad 302 (FIG. 3) placed beneath a standard pillow 304 in apillow case. When a user rests his/her head 308 on pillow 304, weight ofthe head is detected 402 (FIG. 4) by pressure sensor 116, and breathingsounds from head 308 and chest 310 are detected by microphones 108, 110,112, 114; pressure detection signals and breathing sound signals frompressure sensor 116 and microphones 108, 110, 112, 114 trigger wakeupcircuit 124 to arouse processor 102 from a standby state. In aparticular embodiment, wakeup circuit 124 arouses or wakes up theprocessor upon detecting pressure on pressure sensor 116. In analternative embodiment, wakeup circuit 124 arouses processor 102 from astandby state upon detecting a combination of pressure on pressuresensor 116 and sounds on microphones 108, 110, 112, or 114 that exceed asound-level threshold.

Once the processor is wakened from the standby state, an audio samplermodule 502 (FIG. 5) of the firmware 106 running on processor 102 usesADC 130 to digitize and record 404 audio in pulse-code modulation (PCM)time-domain form, in some embodiments firmware 106 includes digitalaudio filtering 504 and downsampling before further processing. A FFTmodule 506 of firmware 106 then performs 406 a fast Fourier transform(FFT) 406 to transform the PCM time-domain audio into a frequency-domainrepresentation and an e-GRU simulated neural network module 508 extracts408 features from the frequency-domain representation and in someembodiments from also the PCM time-domain audio. A neural networkclassifier 510 then detects 410 candidate events and sounds of interest,if no such candidate events and sounds of interest are detected in atimeout interval the processor 102 may revert to a low-power state afteruploading any data in data store 132.

Candidate events as classified, together with the features on whichtheir classification was based by classifier 510 and a timestamp fromclock timer 136, are stored 412 in data store 132; since features arestored but not PCM audio or the frequency-domain representation ofaudio, data recorded during speech is generally unintelligible to alistener. Periodically, data, including timestamps, candidate events,and the features on which the candidate events are based, in data store132 are uploaded 414 by a short-range digital radio driver module 514 offirmware 106 using digital radio 134 to a smart device 204 (FIG. 2) suchas a smartphone or other BAN hub, which then relays 416 the data over anetwork, which may be the internet, to server 210. Code 212 of server210 then reclassifies 418 candidate events based on the features onwhich candidate events were based into final detected events. Inembodiments, the final detected event classifications include normalinhalation and exhalation breathing sounds, crackles, wheezes, coughs,snoring, gasping and choking, normal heart sounds, tachycardic heartsounds, speech, and arousal. Code 212 also compiles statistics basedupon the detected events including respiratory rate, heart rate, andinspiratory-expiratory ratio as well as frequency of wheezes and coughs.

Code 212 enters 420 both detected events and statistics based on thedetected events into database 214. When accessed by a smartphone, tabletcomputer, or a workstation running either a web browser or a specificapplication, code 212 provides 422 the events and statistics fromdatabase 214 to users.

In an embodiment, events in PCM time-domain form and frequency domainrepresentation are classified 410 to detect candidate events bycandidate classifier 510 by a e-GRU neural-network classifier trained ona large dataset of sounds previously classified manually. The e-GRUneural network, is an embedded Gated Recurrent Unit (e-GRU) is based onGRUs as defined in Cho, K., Bengio, Y., et al (2014). Learning PhraseRepresentations using RNN Encoder-Decoder for Statistical MachineTranslation. In Proc. EMNLP 2014. The e-GRU is redesigned to meet theresource constraints of low power micro-controllers with three keymodifications: 1) a single gate mechanism, 2) 3-bit exponential weightquantization and 3) solely fixed-point arithmetic operations. Thesemodifications lead to enormous reductions in memory and computationrequirements in e-GRU compared to prior GRUs.

A single gate mechanism modification for the GRU leads to a significantreduction in parameters and computation. An e-GRU cell definition isprovided below.

-   -   z_(t)=softsign(W_(z)⊙[h_(t-1), x_(t)]) update gate    -   h_(t)=(1−z_(t))*h_(t-1)+z_(t)*h _(t) cell state    -   z_(t)=(softsign(W_(z)⊙[h_(t-1), x_(t)]) update gate    -   {tilde over (h)}_(t)=(softsign(W_(h)⊙[h_(t-1), x_(t)])+1)/2        candidate state    -   h_(t)=(1−z_(t))*h_(t-1)+z_(t)*{tilde over (h)}_(t) cell state

Besides the singe gate mechanism, the e-GRU also employs an aggressive3-bit exponential quantization for the weights. Weight quantization isnot new and previous studies have demonstrated that low precision neuralnetworks perform well [4]. In practice, however, 8-bit quantization istypically used in low resource applications. In [5], it was found thatwhile binarization of weights hurts the performance of GRU models,whereas a form of exponential quantization, pow2-ternarization, onlysuffers a small reduction in accuracy. In e-GRU, we explored thisfurther by investigating exponential quantization in tandem with thesingle gate optimization. We found that using septenary weights (3-bits,7 levels) was effective for e-GRU. Furthermore, since the quantizedlevels were negative integer exponents of 2, this process eliminated theneed for weight multiplications (bit shifting is used instead)drastically reducing computation time of an e-GRU cell in processorslacking multiply hardware. A single e-GRU cell requires only 2 bytes ofmemory, 12 times smaller than needed for a full precision GRU.

Finally, e-GRU uses fixed point arithmetic for fast execution on lowpower microcontrollers that have no hardware Floating Point Unit. Wefound the Q15 fixed point format was effective. All operations withinthe e-GRU network are undertaken in the Q15 format. For the activationfunctions, integer approximations to the softsign are used which featureleft-shifts, additions and division in Q15. As alluded to above, weightmultiplications are performed using left-shift operations since allweights are negative integer exponents of 2. Intermediary products areclipped to remain in Q15 format. The summation of multiple e-GRU nodesis, however, allowed to overflow to 32 bits since it is constrained to[−1,1] range by the Q15 activations that ensue. From our simulations, wediscovered that all inputs to an e-GRU network flow through the entiremodel in Q15 format and result in an output that is precise to at least2 decimal places compared to those from an equivalent full precisionnetwork.

In all, e-GRU performs comparably with GRU and thus can enable robustacoustic event detection on an ultra-low power wearable device.

Combinations

The features herein described may be combined in various ways inembodiments anticipated by the inventors. Among embodiments anticipatedby the inventors are:

A device designated A configured for monitoring physiological soundsincludes at least one microphone coupled to an analog-to-digitalconverter (ADC); a digital radio; and a processor configured withfirmware in a memory. The firmware includes machine readable code forusing the ADC to digitize audio from the at least one microphone intodigitized time-domain audio, performing a fast Fourier transform on thedigitized time-domain audio to provide frequency-domain audio, executinga first neural network on the digitized time domain audio and thefrequency-domain audio to extract features from the audio and at leastone pressure sensor, executing a classifier on the features to identifycandidate events, and using the digital radio to upload the candidateevents and features. The at least one pressure sensor is coupled toawaken the processor from a low-power state.

A device designated AA including the device designated A wherein thefirst neural network is an embedded Gated Recurrent Unit (e-GRU) havingweights trained to extract features of use in the classifier.

A device designated AB including the device designated A or AA whereinthe classifier is a second neural network.

A device designated AC including the device designated A, AA, or ABwherein the at least one pressure sensor is coupled to awaken theprocessor through a wake-up circuit, the wake-up circuit also coupled tothe at least one microphone.

A device designated AD including the device designated A, AA, AB, or ACwherein the at least one microphone is a plurality of microphones.

A device designated AE including the device designated A, AA, AB, AC, orAD wherein the candidate events comprise normal inhalation andexhalation breathing sounds, crackles, wheezes, coughs, and snoring.

A device designated AF including the device designated A, AA, AB, AC,AD, or AE wherein the candidate events comprise gasping, choking, andspeech sounds.

A device designated AG including the device designated A, AA, AB, AC,AD, AE, or AF wherein the candidate events further comprise heartsounds.

A device designated AH including the device designated A, AA, AB, AC,AD, AE, AF, or AG embedded within or attached to a pillow.

A system designated B including the device configured for monitoring ofphysiological sounds designated A, AA, AB, AC, AD, AE, AF, AG, or AH;and a smart device such as a smartphone, tablet computer, smartwatch, orBAN hub, configured to receive the uploaded candidate events andfeatures from the digital radio.

A system designated BA including the system designated B furtherincluding code configured to classify and perform statistical analysison the candidate events and features, the candidate events beingclassified into events comprising normal inhalation and exhalationbreathing sounds, crackles, wheezes, coughs, and snoring.

A system designated BB including the system designated BA or B whereinthe smart device is configured to relay the candidate events andfeatures to a server, the code configured to classify and performstatistical analysis on the candidate events and features beingexecutable on a server.

A method of monitoring breathing during sleep designated C includesattaching to a pillow, or embedding within a pillow, a sleep andbreathing monitor device; extracting features from sound recorded withthe sleep and breathing monitor device; classifying the extractedfeatures to detect candidate events; and uploading the candidate eventswith the extracted features and a timestamp.

A method designated CA including the method designated C whereinextracting features from the sound is performed by performing a fastFourier transform to generate a frequency domain representation of thetime domain sound, and using a first neural network on both the timedomain sound and the frequency domain representation to extract thefeatures.

A method designated CB including the method designated CA or C whereinthe classifying is performed with a second neural network.

Changes may be made in the above system, methods or device withoutdeparting from the scope hereof. It should thus be noted that the mattercontained in the above description or shown in the accompanying drawingsshould be interpreted as illustrative and not in a limiting sense. Thefollowing claims are intended to cover all generic and specific featuresdescribed herein, as well as all statements of the scope of the presentmethod and system, which, as a matter of language, might be said to falltherebetween.

What is claimed is:
 1. A device configured for monitoring physiologicalsounds comprising: at least one microphone coupled to ananalog-to-digital converter (ADC); a digital radio; a processorconfigured with firmware in a memory, the firmware comprising machinereadable code configured to: use the ADC to digitize audio signals fromthe at least one microphone into digitized time-domain audio signals,perform a fast Fourier transform on the digitized time-domain audiosignals to provide frequency-domain audio signals, execute a firstneural network on at least the frequency-domain audio signals to extractfeatures from the frequency-domain audio signals and at least onepressure sensor, execute second neural network configured to classifyevents on the features to identify candidate events, and use the digitalradio to upload the candidate events and features; where the at leastone pressure sensor is coupled through a wake-up circuit configured toawaken the processor.
 2. The device of claim 1 wherein the first neuralnetwork is an embedded Gated Recurrent Unit (e-GRU) having weightstrained to extract the features from the frequency domain audio signalsfor the second neural network.
 3. The device of claim 2 wherein thesecond neural network is a second e-GRU neural network.
 4. The device ofclaim 1 wherein the wake-up circuit is also coupled to the at least onemicrophone.
 5. The device of claim 1 wherein the at least one microphoneis a plurality of microphones.
 6. The device of claim 1 wherein thecandidate events comprise normal inhalation and exhalation breathingsounds, crackles, wheezes, coughs, and snoring.
 7. The device of claim 6wherein the candidate events further comprise gasping, choking, andspeech sounds.
 8. The device of claim 7 wherein the candidate eventsfurther comprise heart sounds.
 9. The device of claim 8 being embeddedwithin, or attached to, a pillow.
 10. A system comprising: the deviceconfigured for monitoring of physiological sounds of claim 1; and asmart device configured to receive the uploaded candidate events andfeatures from the digital radio, the smart device being selected fromthe group consisting of a body-area network (BAN) hub, a smartphone, acellular-capable smartwatch, and a tablet computer.
 11. The system ofclaim 10 further comprising code configured to classify and performstatistical analysis on the candidate events and features, the candidateevents being classified into events comprising normal inhalation andexhalation breathing sounds, crackles, wheezes, coughs, and snoring. 12.The system of claim 11 wherein the first neural network is an embeddedGated Recurrent Unit (e-GRU) having weights trained to extract thefeatures for the second neural network.
 13. The device of claim 12wherein the second neural network is a second e-GRU neural network. 14.The system of claim 11 wherein the smartphone is configured to relay thecandidate events and features to a server, the server configured toclassify and perform statistical analysis on the candidate events andfeatures being executable on the server.
 15. The device of claim 2wherein the candidate events comprise normal inhalation and exhalationbreathing sounds, crackles, wheezes, coughs, and snoring.
 16. The deviceof claim 3 wherein the candidate events comprise normal inhalation andexhalation breathing sounds, crackles, wheezes, coughs, and snoring. 17.The device of claim 4 wherein the candidate events comprise normalinhalation and exhalation breathing sounds, crackles, wheezes, coughs,and snoring.
 18. The device of claim 5 wherein the candidate eventscomprise normal inhalation and exhalation breathing sounds, crackles,wheezes, coughs, and snoring.